Time-partitioned Index Design for Adaptive Multi-Route Data Stream Systems utilizing Heavy Hitter Algorithms

Adaptive multi-route query processing (AMR) is a recently emerging paradigm for processing stream queries in highly fluctuating environments. AMR dynamically routes batches of tuples to operators in the query network based on routing criteria and up-to-date system statistics. In the context of AMR systems, indexing, a core technology for efficient stream processing, has received little attention. Indexing in AMR systems is demanding as indices must adapt to serve continuously evolving query paths while maintaining index content under high volumes of data. Our proposed Adaptive Multi-Route Index (AMRI) employs a bitmap time-partitioned design that while being versatile in serving a diverse ever changing workload of multiple query access patterns remains lightweight in terms of maintenance and storage requirements. In addition, our AMRI index design and migration strategies seeks to met the indexing needs of both older partially serviced and newer incoming search requests. We show that the effect on the quality of the index configuration selected based on using AMRIs compressed statistics can be bounded to a preset constant. Our experimental study using both synthetic and real data streams has demonstrated that our AMRI strategy strikes a balance between supporting effective query processing in dynamic stream environments while keeping the index maintenance and tuning costs to a minimum. Using a data set collected by environmental sensors placed in the Intel Berkeley Research lab, our AMRI outperforms the state-of-the-art approach on average by 68% in cumulative throughput.